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THE DIRECTOR OF CENTRAL INTELLIGENCE 

WASHINGTON, D.C. 20505 


Senior Review Panel NIC-05182-84 

10 September 1984 


MEMORANDUM FOR: Chairman, National Intelligence Council 

SUBJECT: Senior Review Panel Examination of Several 

Estimative Products from the First Halves of 
1982 and 1983 


Purpose of Study 

1. In response to your request, the Senior Review Panel 
undertook a sampling of estimative products published by the 
National Intelligence Council during the first six months of 1982 
and the equivalent period of 1983. You asked that the Panel 
address, in particular, two questions: 

How well do the projections stand up in the light of 
subsequent events? 

How useful, in our view, were the papers to the 
policymaking community? 

I 

2. The field of inquiry was thus the four NIEs and 17 SNIEs 
published in the first half of 1982 and the 12 NIEs and 13 SNIEs 
published in the first half of 1983. 

Main Findings 

3. Annexes to this memorandum contain our specific 
comments. Our main summary conclusions are: 

3. *1982-1983 estimative production was of extremely 
high quality in forecast validity and policymaker utility. 

Of the 46 NIEs and SNIEs published in the two .semestral 
periods, 45 met--or are meeti ng--acceptab1y severe standards 
of estimative dependability. 
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I V I fc. V b. 


b. There was only one failed estimate in the two half- 
years-^ 


c. The high estimative performance of 1982 was 
maintained in 1983 with an appreciable increase in quantity. 

d. For the most part, V982 and 1983 production would-- 
in our experience--ha ve met most' policymaker needs in 
timeliness, policy issue relevance, analytical quality, 
readability, and estimative judgment. Actual use made is, 
perhaps, a separate question and one beyond the scope of our 
study. Overall, as policymaking support, we would regard 
two-thirds to three-quarters of the papers as "Superior to 
Outstanding," with the percentage rising as one moves from 
1982 to 1983. 

e. Length of preparation/coordination time bore little, 
if any, relation to quality of finished product. Granted 
special response factors to high priority estimative 
requests, there seemed in these two years an almost inverse 
relation: the quicker the response, the better the product. 

f. Improvements in quality are still--and probably 
always will be--possi bl e. We cite a number of these in the 
concluding section to this memorandum. 

Section of the Sample 

4. As indicated, our summary findings derive from a review 
of all the NIEs/SNIEs published in the two periods. We selected, 
as requested, a smaller sample for more intensive study on the 
basis of the following criteria: (a) the projective time frame 
of each had to be sufficiently short-term to permit a valid 
comparison with our perceptions of subsequent "ground truth" 
twenty-four and twelve months later; (b) the ultimate sample 
needed to be broadly representative of the diversity of NIC 
output and large enough to be statistically significant, taking 
into account the increased production of 1983 over 1982; and 

(c) subsequent Community and Agency analyses were available to 
test validity half-lives. 

5. The sample which emerged comprised the following 12 
NIEs/SNIEs. Our individual comments are attached as case studies 
at the indicated Tabs: 
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Methodol ogy 


6. The estimates themselves were of course our primary 
source. Additionally, we requested computer runs of Agency 
holdings of finished and unfinished intelligence bearing on 
subsequent events and trends. From these we selected for 
detailed study what we believed pertinent to our inquiry, 
numbering from two to three dozen for some estimates to several 
hundred for others. All Panel members reviewed all retrieved 
documents. In some cases we found it necessary, as our work 
progressed, to request papers published prior to the estimates. 
Finally, our judgments necessarily reflect our own recollections 
and impressions of the passage of events since the issuance of 
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the estimates forming the sample. We have tried to discipline 
this subjectivity, as indicated, by the production record. 

Concluding Observations 

7. As indicated above, estimative production in the two 
periods reviewed seemed to us of an impressively high quality. 

Quantity also increased. With few exceptions the estimates were 
timely. They addressed issues of policy importance. They would 
have been, again with a few exceptions, of value to 

deci onmakers . And the majority provided predictions which were 
perceptive and which have been validated by subsequent events. 

8. In our view, it is also true that length remains a 
problem, and that too much of the burden of concision may now be 
carried by the Key Judgments. Within generally excellent 
estimates, individual findings are at times more discursive and 
fuzzy than rigorous thinking would abide. There are persistent 
tendencies to hedge bets without giving the reader a feel for the 
odds on--often implicit or casually devel oped--al ternati ve 
outcomes. And Community practice regarding dissenting footnotes 
on major issues may need a further look. In particular, their 
rationale appears to need fuller development in the text--perhaps 
with an opportunity for a one-round rejoi nder--and additional 
highlighting in the Key Judgments. At the end, clarity of aim, 
brevity of expression, and a decent respect for man's 
imponderables remain the cardinal virtues. 

9. The distribution of this report has been limited as 
noted. 
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Tabs A through L 


cc: DCI 
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